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This brief describes findings from a study of teachers who were rated less than proficient 
on one or more standards on a large urban district’s newly developed educator evaluation 
system. Evaluators prescribed professional development activities, such as courses, 
workshops, coaching, and mentoring, and professional practice activities, such as the 
use of specific instructional strategies, that teachers should undertake to improve their 
practice. For each standard on the evaluation rubric, 9-58 percent of teachers received 
prescriptions that included professional development activities, and at least 97 percent 
received prescriptions that included professional practice activities. Among teachers who 
completed a survey, more teachers reported participating in professional activities related 
to the evaluation system’s two instruction-based standards than in professional activities 
related to the standards focused on family and community engagement and professional 
culture. For all standards, less than 40 percent of teachers participated in all the activities 
evaluators prescribed to them. The percentage of teachers who received at least a 
proficient rating in the curriculum, planning, and assessment standard on their subsequent 
summative evaluation was higher for teachers who participated in activities related to that 
standard than for teachers who did not participate in standards-related activities. 


This brief summarizes the findings of Shakman, K., Zweig, J., Bocala, C., Lacireno-Paquet, N.. & 
Bailey, J. (2016). Teacher evaluation and professional learning: Lessons from early implementation in a 
large urban district (REL 2016-115). Washington, DC: U.S. Department of Education, Institute of 
Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional 
Educational Laboratory Northeast & Islands. That report is available at http://ies.ed.gov/ncee/edlabs/ 
projects/project.asp?ProjectID=3461. 
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Why this study? 


Policymakers and researchers recommend that data collected through educator evaluation be used to guide 
teachers’ continuous professional improvement (for example, Coggshall, Rasmussen, Colton, Milton, & 
Jacques, 2012; Curtis & Weiner, 2012; Danielson, 2012; Darling-Hammond, 2012; Goe, Holdheide, & 
Miller, 2014). However, research has yet to determine whether and how this evaluation-based improve- 
ment process is happening in school districts. This study, by Regional Educational Laboratory Northeast & 
Islands in collaboration with the Northeast Educator Effectiveness Research Alliance, looked closely at one 
large urban district’s educator evaluation system from May 2013 to May 2014. It also examined the written 
feedback evaluators provided to teachers who were rated less than proficient in one or more standards on 
the evaluation rubric (box 1). This written feedback, which the district refers to as prescriptions,! provides 
information about how teachers may improve through professional activities, whether participating in pro- 
fessional development, such as a course or coaching, or engaging in professional practice activities, such as 
submitting documents or implementing an instructional strategy. 


This brief focuses on these prescriptions and presents findings for the following research questions: 

e In what standards did teachers receive prescriptions? 

e For each standard, what professional activities did evaluators prescribe? 

e What professional activities did teachers report they had participated in? 

e Did the professional activities in which teachers participated align with the standards for which 
they received prescriptions and the activities that were prescribed? 

e Among teachers who participated and those who did not participate in any professional activi- 
ties in the standard in which they had a prescription, what percentage improved their summative 
ratings at the standard level from May 2013 to May 2014? Among teachers who participated and 
those who did not participate in the specific types of professional activities prescribed by their 
evaluators, what percentage improved their summative ratings at the standard level from May 2013 
to May 2014? 


Box 1. Educator evaluation in the study district 


In the district’s educator evaluation system, teachers are assessed using a rubric that defines four standards 
of effective practice: 

¢ Curriculum, planning, and assessment (standard 1). 

¢ Teaching all students (standard 2). 

e Family and community engagement (standard 3). 

e Professional culture (standard 4). 


If the evaluator (the principal or designated staff member) rates the teacher as “needs improvement” or “unsat- 
isfactory” on any of the four standards, the evaluator prescribes professional activities. Specifically, the evalua- 
tor electronically submits one or more prescriptions for the teacher. Each prescription, which may be submitted 
at any time during the school year, indicates the following: 

e The standard in which the teacher is expected to improve. 

e A problem statement that outlines the issue that has been identified. 

e Evidence associated with the issue. 

e Prescribed activities for the teacher. 


Fulfilling a prescription is not required for a teacher’s rating to improve; it is one of several factors that an evalu- 
ator might consider in making rating decisions. 


How the study was conducted 


The data for the study consisted of teacher evaluation ratings, prescriptions, and characteristics for 586 
teachers who received a “needs improvement” or “unsatisfactory” rating in one or more standards on their 
2013 summative or 2013/14 school year formative evaluations, received a prescription between May 2013 
and February 2014, and did not have an overall rating of unsatisfactory on their 2013 summative evalua- 
tion.’ (The 2013 summative evaluation was conducted in May 2013, and the 2013/14 formative evaluation 
was conducted by February 2014.) The study team calculated the percentage of the full population that 
received prescriptions for each standard. 


For a random sample of one-quarter of the teachers, stratified by age and race/ethnicity, the study team 
coded the activities in each prescription into nine types within two broad categories of professional activi- 
ties: professional development and professional practice. Professional development activities are traditional 
professional development (such as courses and workshops) and integrated professional development (such 
as coaching, meeting with an evaluator, and observing colleagues). Professional practice activities are inde- 
pendent work to improve a teacher’s practice through instructional or other professional strategies in or 
outside the classroom. Professional practice activities are not typically conducted through interaction with 
other professionals.’ 


A survey was administered to the same population of teachers to capture the types of professional activities 
in which teachers participated during the evaluation cycle (May 2013—May 2014) related to each of the 
four standards on the evaluation rubric. Forty-two percent of eligible teachers completed the survey, so the 
findings should be interpreted cautiously. The survey respondents were more likely than the full population 
of teachers with prescriptions to be older, White, and female (see appendix C of Shakman, Zweig, Bocala, 
Lacireno-Paquet, and Bailey, 2016, for more information about the nonresponse analysis). 


The study team combined survey and prescription data to assess whether teachers took the prescribed 
actions and then calculated the percentage of teachers whose 2013/14 summative ratings were at least pro- 
ficient, grouped by whether they participated in activities for the standards in which they received a pre- 
scription and by whether the activities aligned with those their evaluators prescribed. 


In addition, interviews were conducted with six teachers and four principals about their experiences with 
the prescription process. 


What the study found 


This section summarizes findings for each of the research questions. 


Teachers received prescriptions across all four standards, and prescriptions tended to include one or two professional 
activities per prescription 


Teachers received prescriptions across all four standards, with 49 percent of teachers receiving a prescrip- 
tion for standard 1 (curriculum, planning, and assessment), 52 percent for standard 2 (teaching all students), 
and 51 percent for standard 3 (family and community engagement). Teachers received the fewest prescrip- 
tions for standard 4 (professional culture), at 34 percent. A majority of teachers in the random sample were 
prescribed just one or two out of the nine professional activity types for standard 2 (61 percent), standard 
3 (92 percent), and standard 4 (66 percent). Only for standard 1 did a majority of teachers receive prescrip- 
tions with three or more types of professional activities (51 percent). 


Teachers received more prescriptions with professional practice activities than with professional development 
activities for all standards 


Across all standards nearly all teachers were prescribed professional practice activities (97-100 percent). By 
contrast, professional development activities were prescribed to 9-58 percent of teachers for each standard 
(table 1). Evaluators most frequently indicated that teachers should practice an instructional strategy, which 
might include the general direction to differentiate instruction or to use a specific student engagement 
strategy such as “accountable talk” or “think, pair, share” (for standards 1 and 2), or that they should prac- 
tice one of the other professional strategies, such as to meet deadlines, arrive at work on time, or follow the 
expectations in the staff handbook (for standards 3 and 4). Some of the least frequently prescribed activ- 
ities in the random sample of teachers’ prescriptions, across all standards, were professional development 
activities such as observing a colleague, meeting with an evaluator, and formal coaching or mentoring. 


More teachers reported participating in professional activities, including both professional development and 
professional practice activities, for standards 1 and 2 than for standards 3 and 4 


More teachers who responded to the survey reported participating in professional activities for standards 
1 and 2 (the instruction-based standards) than for standard 3 (family and community engagement) and 
standard 4 (professional culture). Eighty percent of survey respondents reported participating in any profes- 
sional activities for standard 1 and 68 percent for standard 2, while only 28 percent reported participating 
in any activities for standard 3 and 34 percent for standard 4 (figure 1). 


Within each standard the percentages of teachers who reported participating in professional development 
and professional practice activities were similar. For example, for standard 1, 79 percent of teachers report- 
ed participating in professional development, and 76 percent reported engaging in professional practice 


Table 1. Percentage of teachers in a random sample prescribed each type of professional activity 
for each standard, May 2013-February 2014 


Curriculum, Family and 
planning, and Teaching community Professional 
assessment all students engagement culture 
(standard 1) (standard 2) (standard 3) (standard 4) 
Type of professional activities ‘(eer 4°) (n 70) (n 75) (1 ¢)) 
Professional development 58 51 9 42 
Workshop or course 17 13 0 18 
Meeting with evaluator 12 13 3 4 
Formal coaching or mentoring 
by a nonevaluator 18 16 1 4 
Formal meeting with a colleague 38 30 8 28 
Observation of colleague 8 16 0 2 
Professional practice 97 97 97 100 
Document submission 59 26 55 42 
Reading resources 18 19 1 6 
Instructional strategies 87 93 4 28 
Other professional strategies 13 14 87 76 
None ? a 2 2 


a. Restricted due to small cell size. 


Note: Percentages may not sum to 100 percent because teachers may have been prescribed more than one activity. Based ona 
random sample of 148 teachers. 


Source: Authors’ analysis based on 2013 and 2014 district data on teacher characteristics and evaluation ratings and evaluator 
prescriptions. 


Figure 1. More survey respondents reported participating in any professional activities related to 
the instruction-based standards (standards 1 and 2) than in professional activities related to the 
other two standards, May 2013-—May 2014 


Percent of survey respondents 
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Note: Teachers may have participated in both professional development and professional practice activities. Reported percent- 
ages are weighted for nonresponse; 248 teachers were included in this analysis. 


Source: Authors’ analysis based on 2013 and 2014 district data on teacher characteristics and evaluation ratings, evaluator 
prescriptions, and a teacher survey. 


activities. For all standards the most commonly reported professional activity was participating in a work- 
shop or course. For standards 1 and 2 submitting documents (60 percent and 50 percent) and meeting with 
colleagues (60 percent and 50 percent) were also commonly reported. Across all standards the least com- 
monly reported activity was meeting with the evaluator. For each standard between 20 percent (standard 1) 
and 72 percent (standard 3) of teachers reported not participating in any activities. 


For all standards less than 40 percent of teachers participated in all the activities their evaluators prescribed 


At least 75 percent of teachers who received a prescription for standard 1 (curriculum, planning, and assess- 
ment) or standard 2 (teaching all students) reported participating in some professional activity, regardless 
of whether it was a prescribed activity, that addressed those standards, whereas only 28 percent of teachers 
who received a prescription for standard 3 (family and community engagement) reported participating in 
any professional activity addressing that standard. Forty-four percent of teachers who received a prescrip- 
tion for standard 4 (professional culture) reported participating in some professional activity related to that 
standard (figure 2). 


Evaluators prescribed professional development activities less frequently than they prescribed professional 
practice activities (see table 1). However, for standard 1 the percentage of teachers who reported participat- 
ing in at least one professional development activity that their evaluator prescribed (66 percent) was the 
same as the percentage that reported participating in at least one professional practice activity that their 
evaluator prescribed (66 percent; figure 3). For standard 2 a higher percentage of teachers reported participat- 
ing in at least one prescribed professional development activity (59 percent) than reported participating in at 
least one prescribed professional practice activity (49 percent). The opposite was true for standards 3 and 4; 
a higher percentage of teachers reported participating in at least one prescribed professional practice activity. 


Figure 2. For all standards less than 40 percent of survey respondents reported participating in all 
the activities their evaluators prescribed, May 2013-—May 2014 
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Note: Reported percentages are weighted for nonresponse. 


Source: Authors’ analysis based on 2013 and 2014 district data on teacher characteristics and evaluation ratings, prescriptions, 
and a teacher survey. 


The percentage of teachers who received at least a proficient rating in the curriculum, planning, and assessment 
standard on their subsequent summative evaluation was higher for teachers who participated in activities related to 
that standard than for teachers who did not participate 


Of teachers with a prescription for standard 1 (curriculum, planning, and assessment), 64 percent who 
participated in some activity related to the standard, regardless of whether it was a prescribed activity, 
received at least a proficient rating in standard 1 on their 2013/14 summative evaluation, compared with 
38 percent of teachers who did not participate in any professional activity for standard 1, despite receiving a 
prescription for that standard. Standard 1 is the only standard in which there was a statistically significant 
difference in the percentage that received at least a proficient rating on their 2013/14 summative evaluation 
between teachers who did participate and those who did not participate in activities for the standard in 
which they received a prescription. 


Figure 3. A higher percentage of survey respondents reported participating in at least one professional 
activity prescribed by their evaluators related to the instruction-based standards (standards 1 and 2) 
than in professional activities related to the other two standards, May 2013-—May 2014 
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Note: Reported percentages are weighted for nonresponse. The figure shows the percentage of survey respondents that partici- 
pated in at least one prescribed activity for professional development and for professional practice, by standard. For example, of 
the teachers who received a prescription in standard 1, responded to the survey, and were prescribed professional development 
activities, 66 percent participated in at least one of the prescribed professional development activities. A teacher may have been 
prescribed both a professional development and professional practice activity in the same prescription. The number of teachers 
with prescriptions who responded to the survey was 125 for standard 1, 139 for standard 2, 123 for standard 3, and 75 for stan- 
dard 4. 


Source: Authors’ analysis based on 2013 and 2014 district data on teacher characteristics and evaluation ratings, prescriptions, 
and a teacher survey. 


Discussion 


The study district was in the early phases of implementing a new evaluation system at the time of this 
study. District leadership viewed participation in the study as an opportunity to look closely at the new 
system to examine the types of data the district collected and to consider the support evaluators and teach- 
ers received. While this study neither examines the quality of the feedback teachers received nor makes 
causal claims about the impact of professional activities on teacher practice, the findings may be helpful to 
states and districts that are designing and implementing aligned systems of evaluation and support. 


There are several possible explanations for the lack of alignment between the types of professional activi- 
ties evaluators prescribed and the activities in which teachers reported engaging: 
e Evaluators and teachers may not have agreed about what teachers needed to do to improve their 
practice; therefore, teachers may have chosen to engage in activities different from those prescribed. 
e Evaluators may not have known what types of professional activities were available to teachers and 
therefore did not prescribe the full range of activities in which teachers could—and did—engage. 
e Evaluators may have prescribed activities that were vague or unclear, and therefore teachers did not 
know how to fulfill the prescriptions. 


e Teachers may have lacked incentive to follow the prescription because they were under no direct 
contractual obligation to fulfill the specific terms of the prescription. 
e Teachers may not have had access to the types of professional activities their evaluators prescribed. 


Further research is needed to understand why there was not greater alignment between the professional 
activities prescribed and the activities teachers reported engaging in for each standard. Yet the findings 
suggest that evaluators might benefit from additional support and guidance related to the development and 
monitoring of specific and actionable prescriptions that support improvements in teacher practice. 


Notes 


While teachers are not contractually bound to fulfill the prescriptions assigned to them, the evaluator 
initiates a prescription in conjunction with a less than proficient rating on one of the standards in the 
evaluation rubric. In the event that teachers do not fulfill the recommended actions in a prescription, 
an evaluator may use a less than proficient rating on the same standard in the subsequent evaluation 
to indicate that the prescription has not been addressed. However, if the evaluator observes sufficient 
improvement in a teacher’s practice, despite not participating in the professional activity prescribed, 
the evaluator may rate the teacher proficient on the standard; there is no direct consequence related to 
not fulfilling the prescription if the teacher’s practice improves. However, a consequence for teachers 
who receive two consecutive less than proficient ratings on standards 1 and 2 at the summative evalua- 
tion is that they receive an unsatisfactory rating overall, and more formal action is taken, with a time- 
frame for possible dismissal. For standards 3 and 4 teachers have more time to improve their ratings. 
The study did not include teachers who received an overall summative rating of unsatisfactory, as these 
teachers were on a more specific and targeted improvement plan. 

See table 1 for the nine professional activity types; see Shakman et al. (2016) for an overview of the 
categories, activity types, and examples. 
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